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In cases where e-leaming is used mainly outside the class, a lecture is given by lecture slide and a small test is given 
at the end of the class. Students fill out a problem in a structured notebook by viewing lecture slide materials in the 
e-leaming after a lecture. In cases where e-leaming is used during school hours, the outline of class on each day is 
explained for about 20 minutes using slides. Students then fill out a problem in a structured notebook for about 60 
minutes while viewing lecture slide materials in the e-leaming. Students are required to learn at their own pace with 
the aim of improving their understanding of lecture contents. In the two kinds of blended classes, students plan study 
support systems at the end of the course. Students submit reports and evaluate them mutually. The effects of the two 
blended classes are compared by significant difference tests. The blended classes by the two methods improve 
significantly the knowledge degree of technical terms and the students’ attitude. No significant differences between 
the two methods are observed for the knowledge degree of technical terms and the students’ attitude as a whole. It is 
found that activities useful to improve attitude are almost similar to the two blended classes. 

Keywords: blended learning, e-leaming, stmctured notebook, plan of study support system, peer estimation 

Introduction 

How to learn is different for each student. Perfect media for all students do not exist. It is possible to 
support many and various student learning styles and to deepen understanding by using more than one medium 
(Adachi, 2007; Bersin, 2004; Miyaji, 2009). The use of e-leaming, which cooperates with and expands the class, 
is a useful means to improve the traditional class (Nishimori et al., 2003; Mochizuki et al., 2003). Reading class 
content on the web increases opportunities to review and prepare. It is tied with settlement of the contents to 
leam many times until a learner is convinced. Even if a learner is absent from a class, the class content can be 
reviewed at his/her convenience. 

Recently, e-leaming is being introduced into university classes with the popularization of the Internet 
(Saito & Kim, 2009). Using an e-leaming system, Arakawa, Ueki, and Fuyuki (2004) proposed the way of 
educating under the repeated cycle of preparation between the class and the review, by establishing a problem 
which can be used to obtain programming knowledge. They reported that this way was effective to acquire the 
ability to think logically. Horita, Murakami, and Morishita (2003) developed the class which is practiced by 
combining a mini lecture, a simple web text and drill material offered commercially. They reported that the 
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learning progress ratio for an improvement group of results was significantly higher than one from the others 
when analyzing information in the class. 

Because motivation for e-leaming is difficult and everyone can easily stop participating in e-learning at 
anytime, e-leaming is exposed to the shortage that the finish rate will be low, but the effect is small. As such a 
fault exists in e-learning, it is possible to improve the effect by compensating for other parts when teachers use 
it to complement a lecture and training. A class which combines e-learning and other media with a lecture is at 
present called a blended class (Thome, 2003; Bonk & Graham, 2006; Kitazawa, Nagai, & Ueno, 2008). 

Someone improves a university education in which the problem-solving ability is trained through creating 
thing and evaluation activities. It is necessary to support students to increase the chance of learning, and allow the 
students to prepare and review materials anytime and anywhere as support of a lecture. Miyaji and Yoshida (2005b) 
had practiced the blended class style for the introduction to computer science course since 2004. They developed 
the class which was conducted by combining e-leaming, a stmctured notebook and a small test. An e-leaming 
course uses learning via lecture slides, learning by exercises and mutual learning and evaluations through students 
making teaching materials. It was reported that the class had made their knowledge increase and their 
consciousness improve. Miyaji, Yoshida, and Naruse (2007) also reported that it is possible to improve the effect 
by increasing interactions with the teacher when the questionnaires for the degree of understanding are performed. 

We examined whether there was a difference in the effect when using e-leaming inside and outside of the 
class. In 2005, an elective lecture on the subject of AI (artificial intelligence) was given to the third-year 
university students in the department of information science using slides. A small test was carried out for about 
10 minutes at the end of the class. We developed an e-learning system that has functions of learning by lecture 
slides and exercises designed to improve the students’ understanding at their own pace outside of the class. A 
structured notebook was distributed to students to write the lesson’s contents in it. The e-learning can be used 
to support their activities and also to prepare and review before and after a class respectively. Since students 
use the e-learning mainly outside the class, it called an outside -the-cl as sroom blended class. 

After being in practice for two years, we considered it to be desirable that students learn at their pace by 
using the e-leaming course during the class to improve their understanding. The outline of class was explained 
for about 20 minutes using slides in 2007. Then they were required to write an answer to a problem in the 
structured notebook for about 60 minutes while viewing the lecture slide materials in the e-leaming in a PC 
(personal computer) room. Since students use e-learning mainly inside the class, it is called an 
inside-the-classroom blended class. 

Comparison of the inside-the-classroom blended class with the outside -the-classroom blended class is not 
studied yet. In this paper, we examine whether there is what kind of difference in the effect of both methods 
that we practiced. A significant difference test was conducted to compare the average learning time, the small 
test, terminology awareness, abilities and perception, which were information related to the two kinds of 
blended classes of AI in 2005 and 2007. The class content, the content of the structured notebook and the 
content of the e-learning are explained in section 2 respectively. The results of analyses of the information are 
shown and are considered in section 3. 

Instructional Design and the Class Content 

The class subject that was the target of our research was an AI in the second term of the third year in the 
Department of Information Science at a university. The lecture contents are shown in Table 1. The class was 
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conducted 15 times with each class being 90 minutes long. A lecture plan in 2007 is shown in Table 2. A 
midterm and a final examination were conducted in the 8th and 15th week in order to motivate learning and 
confirm the understanding. We hope that various activities could further deepen the understanding and broaden 
the lecture contents. 

Table 1 



The Number of Teaching Materials in the E-Learning and the Structured Notebook 



Chapter 


Section 


No. of 
sliders 


No. of 
exercises 


No. of problems 
in structured 
notebook 




1.1 Knowledge 


9 


13 


4 




1.2 Information processing in the brain 


7 


6 


6 


1. Artificial intelligence and 


1.3 Artificial intelligence 


7 


9 


6 


knowledge engineering 


1.4 Knowledge engineering 


12 


19 


6 




1.5 Expert system 


8 


10 


3 




1.6 Cognitive science 


6 


3 


3 




2. 1 Kind of knowledge 


9 


20 


8 




2.2 Necessity of knowledge representation 


5 


7 


4 


2. Knowledge representation 


2.3 Semantic network 


14 


16 


11 




2.4 Frame theory 


14 


25 


20 




2.5 Production system 


23 


29 


17 




3.1 Significance of CAI 


7 


16 


8 


3.CAI 


3.2 Construction of CAI 


6 


6 


2 




3.3 Learning pattern of CAI 


28 


23 


17 




3.4 Devices to improve the learning effects of CAI 


5 


2 


1 




4.1 History of intelligence CAI 


9 


9 


9 




4.2 Basic construction of intelligence CAI 


7 


13 


7 




4.3 Knowledge base of teaching materials 


13 


11 


7 




4.4 Student model 


27 


25 


19 




4.5 Tutoring rule base 


11 


10 


8 


4. Intelligent CAI 


4.6 User interface 


8 


4 


2 




4.7 Inference engine 


7 


2 


2 




4.8 Answer matching and diagnosis processing 


9 


3 


2 




4.9 Exercise generation 


10 


5 


4 




4.10 Learning history 


6 


2 


3 




4.11 Question and answer 


16 


3 


3 


Total 




283 


291 


182 



The outside-the -classroom blended class will be explained in the following. Thirty-seven students 
participated in 2005. Students were given a lecture by projecting slides onto a screen in the classroom for about 
70 minutes. We distributed a structured notebook which consists of 182 questions within 34 pages, as shown in 
Figure 1. Students were recommended to use it for preparation and review. A small test which consists of two 
to five questions was conducted while allowing the students to see a textbook and distributed material for about 
10 minutes at the end of the class. The questions asked about points from the lecture. 
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Table 2 



Lecture Plan for AI 



Time 


Lecture contents 


1 


Advance questionnaire, section 1.1, 1.2, 1.3 


2 


Section 1.4, 1.5, 1.6, 2.1 


3 


Section 2.2, 2.3 


4 


Section 2.4, 2.5 


5 


Section 2.5 


6 


Section 4.1, 4.4, 4.5, 4.6 


7 


Explanation for planned study support systems, exercise by e-leaming 


8 


Midterm examination 


9 


Section 5.1, 5.2, 5.3 


10 


Section 5.4, 5.5, 5.6, 5.7 


11 


Section 5.8, 5.9, 5.10,5.11 


12 


Peer assessment for planned study support systems 


13 


Exercise by e-leaming 


14 


Peer assessment for planned study support systems posterior questionnaire 


15 


Final examination 



©Mini lecture by slides 



ir 



i 



©Filling out a 
strutured notebook 
while viewing 
lecture slides in e- 
learning 



©Small test 



©Learning 
by struturei 
notebook 








Submission of re por1 
S ^^ 0Wnl0ad 



I 




©Learning by 
lecture slides 

©Learning by 
exercises 




Home 



ev dlu, 



Won 



sheet 



Teacher 



©Mutual rating of the 
planned study 
support system 

©A report and 
evaluation sheet 

© Viewing answers of 
the small test 

©Bulletin board 



room 



Figure 1. An example of the contents in the structured notebook. 



Students were required to bring the structured notebook each time they came to class. They wrote answers 
for one question in it to establish the class contents in each class for about ten minutes. A teacher went around 
the desks while the students were writing and responded to their questions. In some cases, how to solve 



problems was explained on the blackboard. 

E-learning has functions of learning via lecture slides and exercises. This can be used for preparation and 
review before and after a class. As AI is an elective subject and involves a lot of technical terms, there are a lot 



of students who find it difficult to understand the contents in only one lecture. At the students’ convenience, 
lecture slides can repeatedly be seen as many times as necessary to improve their understanding. Individual 
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learning is conducted by using such e-learning methods. Students were required to solve problems using 
e-learning mainly in the classes one week prior to having examinations in order to encourage solving problems 
outside the classroom by e-leaming. It is our goal that all the students acquire subject content by such means. 

A student was required to plan a study support system in order to utilize the knowledge learned concerning 
the subject contents at the end of the course. The handouts for several examples of study support systems were 
distributed and explained. In the ninth week, two page structure sheets were distributed to help plan a study 
support system. The report sheets were registered after submission. Students saw and rated them mutually. 
Then, they corrected and improved the plans by referring to the evaluations and teachers’ advice. These 
activities encouraged interaction among the students. 

After completing the e-leaming, students were required to input their learning situation, evaluation value of 
the learning contents, degrees of abilities and attitude towards an evaluation sheet of Excel, and submit the file. 
We told the students to include all submission materials in their personal records to create motivation for learning. 

The inside-the-classroom blended class will be explained in the following. There were 29 participants in 
the 2007 program. Diagram of the blended class in 2007 is shown in Figure 2. The only instmctional design 
difference from the class of AI in 2005 is explained as follows. The class was conducted in a PC room 14 times. 
The outline of the lesson for each day was explained for about 20 minutes using slides. Then students 
completed problems in their stmctured notebook for about 60 minutes while reading lecture slide materials by 
the e-learning. A student learned in a class using mainly e-learning in this way. Students were allowed to leam 
at their own pace to improve their understanding. While students filled out their notebook, we inspected their 
work from between desks and responded to their questions accordingly. As the answers to the small test could 
be found within the e-learning, we encouraged the students to check it. 

The inside-the-classroom blended class includes mini lecture, e-learning, structured notebook, planning a 
study support system and peer assessment for the report. We devised like this so that a greater synergy was 
brought through supplementing these media mutually. 
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Figure 2. Diagram of the blended class using the e-leaming in the lesson. 
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Figure 3. An example for contents of e-learning. 

Results of the Analysis and Discussion 



Information related to the classes in 2005 and 2007 was used to compare the effects of the blended classes 
where the e-learning was used both inside the class and outside the class. We know the difference in the 
learning situations from a significant difference test on the average learning time and average number of pages 
completed in the e-leaming in 2005 and 2007. We know the difference in the amount of knowledge acquired 
from the difference in the average test scores for the common problems from the small test in 2005 and 2007. 
The knowledge degree obtained by questionnaires concerning technical terms on AI was measured twice, once 
in advance (first week), and once posteriori (14th week). The difference for growth of the knowledge degree 
was confirmed through a significant difference test. The questionnaires for abilities and attitude were conducted 
twice, once in advance (first week) and once posteriori (14th week). We know the effect on the attitude by 
analyzing the change in the rating value. We calculated the difference in the effect by a significant difference 
test between the changes in 2005 and 2007. 

The signs ***, ** and * in the test result show that significant differences are observed at 0.001, 0.01 and 
0.05 significance levels respectively in the following. A presence of a significant difference is judged by 
significance level 0.05. 

Comparison of Learning Times 

The learning history of e-learning in 2005 and 2007 was calculated. The time of slide learning and 
exercise learning, and the number of their pages are shown in Table 3. There were 31 students in 2005 and 28 
students in 2007 who used the e-leaming course. A significant difference was observed for the average time of 
the slide learning in 2005 and 2007 (p < 0.001). It was found that the average time in 2007 was longer than that 
in 2005. Since the threads of lecture contents were gathered by entering answers for problems in the structured 
notebook while viewing a lecture slide during the class in 2007, the time of slide learning became about 6.4 
times as long as that in 2005. A slide in 2005 and in 2007 was viewed on average 0.28 times and 1.08 times 
respectively. Viewing time of the slides in 2005 and in 2007 was on average 44 seconds and 73 seconds 
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respectively. We think it is necessary to view the lecture slides more than twice to understand the contents. The 
number of viewing slides in both 2005 and 2007 was quite low. We guess that after the structured notebook is 
filled up, student feel that it is not necessary to view the slides, and they stopped viewing the slides. 

Table 3 



The Learning Time and the Number of Pages for Each Learning Topic 



Learning content 




2005 


2007 




T-test 


m 


SD 


m 


SD 


t 


Results 


Learning time of slides (sec.) 


3,482 


4,768 


22,401 


7,070 


11.1 


*** 


No. of pages learned sliders 


79.0 


89.6 


305.6 


148.9 


6.5 


*** 


Learning time of exercises (sec.) 


7,819 


4,552 


5,061 


2,724 


2.5 


* 


No. of pages learned exercises 


193.0 


109.9 


190.6 


95.5 


0.1 





Notes. *** p < 0.001; * p < 0.05. 



On the other hand, it was admitted that the time of the exercise learning in 2005 was significantly longer 
than that in 2007 at a 1% of significant level. There was no significant difference between the number of pages 
in 2005 and 2007. This shows that it took a longer amount of time for a student in 2005 to answer one exercise 
than that needed in 2007. When both levels of understanding were the same, it means that the students in 2007 
were in a more desirable state than the students in 2005, as they could solve the exercises in a shorter time than 
the students in 2005. However, as each of the exercises was solved only about 0.66 times and 0.65 times on 
average, students have not yet solved enough pages of the exercises to fully understand the lecture content. 

Comparison Between the Scores of the Small Tests 

The number of questions in the small test in 2005 and 2007 were 44 and 39 respectively. The average 
score for the 36 common exercises was compared. The average score and the standard deviation of the small 
test were (76.0 and 8.2) in 2005 and (81.6 and 5.4) in 2007 respectively. A significant difference between these 
could be seen at a 1% significance level as t = 2.90 (p < 0.01). This result shows that the average score of the 
small test in 2007 was higher than that in 2005. 

Comparison of Changes Between Basic AI Knowledge Levels 

A questionnaire consisting of 50 technical terms was conducted twice, once in advance and once posteriori 
to discover the basic knowledge level for AI (Nakamura, 2004; Miyaji & Yoshida, 2005a). The numbers of 
respondents for the tests, both in advance and posteriori, were 30 in 2005 and 25 in 2007 respectively. 

The knowledge degree was rated on a three -point scale as follows: (1) I do not know; (2) I do not know in 
detail, but I heard; and (3) I know. In 2005, the average of knowledge in advance and posteriori was 1.5 and 2.4 
respectively. In 2007, the average degree of knowledge in advance and posteriori was 1.8 and 2.5 respectively. 

Significant differences were observed for the average degree of all 50 terms at a 0.1% significance level in 
both years by Wilcoxon signed-rank test as shown in Table 4. This indicated that the amount of knowledge 
increased as the degree of knowledge of the technical terms increased as a whole. The results of the Wilcoxon 
signed-rank test are shown in Table 4 for the 50 terms between in advance and posteriori from year-on-year. 
The terms are sorted in descending order of test statistics value in 2007. Significant differences were observed 
for 40 terms in 2005 and 42 terms in 2007 at a 5% significance level respectively. These indicated that the 
knowledge for almost all terms was increased in both years. 
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Table 4 



Change in the Degree of Knowledge 









2005 








2007 








d 


Vd 


z 


Result 


d 


Vd 


z 


Result 


Inheritance of property 


0.8 


0.30 


4.4 




0.7 


0.54 


4.4 






Inference engine 


1.4 


0.46 


4.5 


*** 


1.3 


0.69 


4.3 


++-i 




Knowledge-based system 


1.0 


0.59 


4.9 


... 


0.8 


0.61 


4.3 






Expert system 


0.8 


0.76 


4.4 




0.8 


0.54 


4.3 






Blackboard model 


0.7 


0.29 


4.7 




0.6 


0.64 


4.2 






Production rule 


0.7 


0.55 


4.8 


*** 


0.5 


0.37 


4.2 






Student model 


0.2 


0.37 


4.3 


*** 


0.3 


0.70 


4.1 






Method of answer matching 


0.1 


0.37 


4.6 




0.2 


0.46 


4.1 






Semantic network 


0.9 


0.75 


4.8 


*** 


0.7 


0.55 


4.1 






Knowledge base 


1.5 


0.40 


4.6 


... 


1.3 


0.61 


4.0 






Frame-based representation 


0.6 


0.60 


4.7 




0.5 


0.33 


4.0 






Knowledge for user interfaces 


0.1 


0.40 


4.3 


*** 


0.2 


0.30 


4.0 






Teaching expertise 


0.1 


0.30 


4.3 




0.3 


0.37 


4.0 






Intelligent CAI 


0.6 


0.60 


4.3 




0.3 


0.46 


4.0 






Exercise generation 


0.3 


0.55 


4.0 




0.3 


0.56 


3.9 






Knowledge base of teaching materials 


0.5 


0.67 


4.5 




0.3 


0.61 


3.9 






Acquisition of knowledge 


0.6 


0.59 


4.1 




0.9 


0.66 


3.9 






Knowledge expression 


1.0 


0.31 


4.0 




0.9 


0.64 


3.9 






Knowledge processing system 


1.0 


0.31 


4.3 




1.0 


0.80 


3.9 






Guidance strategy 


0.2 


0.58 


4.2 




0.3 


0.58 


3.8 






Decision support 


1.1 


0.41 


4.7 




1.1 


0.46 


3.7 






Learning history 


0.2 


0.53 


3.5 




0.1 


0.46 


3.7 






Factual knowledge 


0.6 


0.38 


4.4 




0.7 


0.50 


3.7 






Procedural knowledge 


1.0 


0.45 


4.3 




0.7 


0.55 


3.6 






CAI courseware 


0.8 


0.65 


3.9 




0.4 


0.47 


3.6 






Question and answer method 


-0.1 


0.13 


3.7 


*** 


0.0 


0.04 


3.6 






Kind of bug 


0.0 


0.45 


4.7 




0.1 


0.37 


3.5 






Declarative knowledge 


0.9 


0.40 


4.5 




0.6 


0.37 


3.5 






Use of knowledge 


0.9 


0.44 


4.0 




0.8 


0.55 


3.5 






Deductive reasoning 


1.0 


0.83 


4.4 




1.1 


0.66 


3.5 






Intelligent computers 


1.2 


0.42 


4.4 




1.3 


0.62 


3.4 






Knowledge engineering 


1.0 


0.79 


4.3 




0.8 


0.43 


3.4 






Artificial intelligence 


1.0 


0.72 


3.2 




0.9 


0.53 


3.3 






Resolution principle 


1.0 


0.55 


4.5 


*** 


0.9 


0.58 


3.2 






Inductive inference 


1.1 


0.48 


3.9 


*** 


1.1 


0.59 


3.1 






Heuristics 


0.8 


0.35 


4.4 


*** 


0.8 


0.65 


2.7 






Problem-solving 


1.1 


0.58 


0.5 




1.3 


0.54 


2.5 






Expertise 


1.4 


0.39 


2.1 




1.5 


0.71 


2.5 






Neuron 


1.0 


0.59 


3.8 


*** 


1.2 


0.60 


2.4 






CAI 


0.3 


0.41 


3.3 


** 


0.4 


0.48 


2.2 






Presumption 


1.0 


0.52 


1.4 




1.1 


0.58 


2.1 






Learning 


1.0 


0.52 


1.0 




1.0 


0.49 


2.0 






Intelligence 


1.0 


0.72 


1.7 


+ 


1.1 


0.76 


1.6 






Database 


0.1 


0.48 


1.9 


+ 


0.5 


0.63 


1.6 






Inference 


1.4 


0.45 


2.9 


** 


1.3 


0.47 


1.6 






PC 


1.1 


0.51 


1.0 




1.5 


0.30 


1.3 






Information 


1.2 


0.51 


1.2 




1.3 


0.36 


1.2 






Memory 


1.1 


0.58 


0.3 




1.2 


0.63 


1.1 






Knowledge 


1.2 


0.65 


1.7 


+ 


1.4 


0.37 


0.9 






Data 


1.3 


0.29 


0.9 




1.4 


0.36 


0.9 






Average 


0.8 


0.16 


4.6 




0.8 


0.12 


4.3 







Notes . *** p < 0.001; ** p <0.01; * p < 0.05; + p < 0.1. 



The growth, which is equal to the degree of knowledge posteriori minus that in advance, is calculated. The 
Wilcoxon signed-rank test was conducted for the average growth of all 50 terms in order to find the difference 
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between 2005 and 2007. The significance difference was not observed for the average growth of all terms 
between both years. 

The Wilcoxon signed-rank test between the average growth degree of knowledge in 2005 and 2007 was 

* 

conducted for each term. Significant differences were observed for only two terms: Neuron (z = 2.1 , p < 0.05) 
and inheritance of property (z = 2.0*, p < 0.05). The significant differences were not observed for the other 48 
terms. Almost exactly the same growth was obtained in both years. 

Comparison of Change in Abilities and Attitude 

The questionnaire, which consisted of about 30 items, including abilities and attitude as shown in Table 5, 
was conducted twice, once in advance and once posteriori. The items in Table 5 are sorted in descending order 
of their t-test statistic values in 2007. Students rated their abilities and attitude on a 9-point scale (superiorly 
demonstrated, 9; strongly demonstrated, 7; somewhat demonstrated, 5; slightly demonstrated, 3; failure to 
demonstrate, 1). Characters of d, Vd and t 0 in Table 5 represent the average difference between the scores in 
advance and posteriori, the sum of the squared deviations, and t-values respectively. The number of 
respondents for the t-tests, both in advance and posteriori were 15 in 2005 and 28 in 2007 respectively. A 
significant difference was observed at a 5% significance level in 2005 and at a 0.1% significance level in 2007. 
We found that the abilities and attitude increased as a whole in both years. The t-test was conducted in order to 
compare the average value for each item in advance and posteriori. Significant differences for 16 items in 2005 
and for 27 items in 2007 were observed at a 5% significance level. 

The t-test for growth, which is equal to the rating values posteriori minus that in advance, was conducted 
in order to find the difference between 2005 and 2007 for all 30 items. No significant difference between 2005 
and 2007 was observed about the growth for all 30 items. The t-test for the growth was conducted for each item. 
Significant difference was observed for only one item (11) at a 5% significance level. The ability to analyze 
information in 2007 seemed to have improved more than that in 2005 as shown in Table 5, but for both years, 
the same growth of the abilities and attitude for all the other 29 items were obtained. 

Comparison of Activities Useful for the Improvement of Abilities and Attitude 

Among 36 activities, students were required to enter some activities useful for the improvement of their 
abilities and attitude in the right column of the rating on the 14th week. The number of activities filled out was 
564 in 2005 and 354 in 2007 respectively. 

The activities useful for the abilities and attitude were totaled by making a cross table. There are many 
cells with less frequency than five in the table. The cluster analysis was conducted about the table using 
Ward’s method with activity as a variable, with the attitude as a case. As a result, the attitude was classified 
into the next four groups I, II, III and IV. Two kinds of numbers are distinguished as explained in the 
following. A number of the attitudes are expressed with a parenthesis number belonging to it. A number of 
the activities are expressed with a number around it. 

The group I consists of (5), (6), (7), (8), (9), (20), (21), (22), (24), (25) and (26). These numbers are the 
attitude shown in Table 5. It was referred to as attitude about “problem-solving and sense of fulfillment”. The 
group II consists of (10), (11), (12), (13), (27), (28) and (29). It was referred to as attitude about “thought, 
creation and expression”. The group III consists of (14), (15), (16), (17), (18), (19), (23) and (30). It was 
referred to as attitude about “communication and evaluation”. The group IV consists of (1), (2), (3) and (4). It 
was referred to as attitude related to “computer”. 
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Table 5 



Significant Test About the Abilities and Attitude 



Items 






2005 








2007 




d 


Yd 


to 


Results 


d 


Vd 


to 


Results 




(11) Ability to analyze information 


0.9 


3.9 


1.8 


+ 


1.8 


2.0 


6.7 


*** 


(13) Ability to express by a means other 
than sentences 


1.3 


2.4 


3.2 


** 


1.7 


1.9 


6.7 


*** 


(6) Ability to map out 


1.3 


2.7 


3.2 


** 


1.8 


2.6 


6.1 


*** 


(2) Understanding of computers 


0.9 


1.9 


2.6 


* 


1.3 


1.6 


5.5 


*** 


(12) Ability to express self-opinions 
with sentences 


1.3 


2.5 


3.1 


** 


1.7 


3.3 


5.1 


*** 


(15) Ability to give a presentation 


1.1 


4.2 


2.0 


+ 


1.8 


3.8 


4.9 


*** 


(5) Ability to clarify problems 


1.2 


2.9 


2.7 


* 


1.4 


2.4 


4.8 


„* 


(29) Ability to create something 


1.2 


5.5 


2.0 


+ 


1.5 


2.8 


4.8 




(17) Ability to communicate with others 


1.2 


2.9 


2.7 


* 


1.7 


4.0 


4.5 


*** 


(22) Ability to complete research 


1.1 


2.8 


2.6 


* 


1.5 


3.2 


4.5 


*** 


(10) Ability to sort information and 


0.9 


2.5 


2.3 


* 


1.1 


1.7 


4.4 


*** 


necessary data 

(28) Ability to think by oneself 


0.7 


3.8 


1.3 




1.2 


2.2 


4.4 


•» 


(4) Methods of computer use 


0.7 


2.8 


1.7 




1.5 


3.5 


4.2 


*** 


(27) Ability to compose knowledge 


1.3 


3.5 


2.8 


* 


1.2 


2.5 


4.1 


*** 


(16) Ability to understand others’ 
explanations 


0.9 


2.8 


2.0 


+ 


1.4 


3.1 


4.1 


*** 


(26) Ability to solve one’s own problems 


1.3 


4.8 


2.2 




1.2 


2.5 


4.1 




(7) Deepening of understanding knowledge 


1.3 


3.7 


2.7 


* 


1.4 


3.4 


4.0 


*** 


(24) Sense of fulfillment 


1.2 


1.2 


4.3 


*** 


1.3 


3.2 


4.0 


*** 


(14) Ability to create simple explanations’ 


0.5 


2.6 


1.1 




1.3 


3.8 


3.6 


** 


(25) Sense of accomplishment 


1.2 


1.6 


3.7 


** 


1.1 


2.9 


3.4 


*** 


(18) Ability to accurately judge 
self-opinions 


0.7 


3.1 


1.6 




1.2 


3.5 


3.3 


** 


(20) Ability to change self-opinions 


0.9 


2.6 


2.1 


+ 


0.8 


2.0 


3.1 




(8) Ability to study independently 


1.4 


2.5 


3.4 


** 


1.1 


3.6 


3.0 


** 


(19) Ability to accurately judge 
others’ opinions 


0.3 


2.1 


0.7 




0.8 


2.2 


2.8 


** 


(23) Ability to work and study 
cooperatively 


0.9 


4.0 


1.7 




1.1 


4.2 


2.8 


* 


(21) Ability to perform detailed 
investigations 


1.5 


5.1 


2.5 


* 


0.9 


3.0 


2.6 


* 


(9) Ability to collect information 


0.5 


4.6 


1.0 




0.7 


2.6 


2.3 




(3) Technical skills with computers 


0.7 


3.8 


1.5 




0.6 


2.4 


2.0 


+ 


(30) Interest in artificial intelligence 


0.9 


1.8 


2.7 


* 


0.1 


2.9 


0.4 




(1) Interest in computers 


0.5 


1.1 


1.9 


+ 


-0.1 


2.3 


-0.3 




Average 


1.0 


2.0 


2.8 


* 


1.2 


1.3 


5.7 


*** 



Notes . *** p < 0.001; ** p < 0.01; * p < 0.05; + p < 0.1. 



The cluster analysis was conducted about the same table using Ward’s method with attitude as a variable, 
with the activity as a case. As a result, the activity was classified into the next five groups. The group 1 consists 
of 17 activities related to a question and learning in respect to lecture. The group 2 consists of four activities 
related to the study support system. The group 3 consists of four activities related to learning by the 
examination and the e-leaming. The group 4 consists of three activities related to the notebook and the report. 
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The group 5 consists of three activities related to attending a lecture and using the PC. 

The frequency is added up for every cluster about the attitude and the activity. Results of 2005 and 2007 
are shown in the left columns of Tables 6 and 7, respectively. Each expected frequency of cell in Tables 6 and 7 
was more than six. Therefore, this table was considered to be 4 x 5 contingency table. The % test was 
conducted for these two tables. As a result, the frequency deflection was recognized for both years (% 2 (i 2 ) = 
223.9, p < 0.001; x 2 (i 2 ) = 99.1, p < 0.001). Therefore, a result of residual analysis is shown in the lower left 
columns of Tables 6 and 7. Significant deflection is shown by a mark in the cell of the lower right columns 
of Tables 6 and 7. Activity useful for improving the attitude is explained by significant cells with greater 
frequency in the following. 

Table 6 



X 2 Test and Residual Analysis About Clusters of Attitude and Activities in 2005 





Observed frequency 


Observed frequency 


Clusters of attitude 
and activities 


1. 

Activities 

question 

and 

learning 
related to 
the class 


2. 

Activities 
related to 
study 
support 
system 


3. 

Activities 
related to 
examination 
and learning 


4. 

Activities 
related to 
structured 
notebook 
and report 


5. 

Activities 
related to 
attending a 
class and 
PC 

utilization 


Sum 


1. 

Activities 

question 

and 

learning 
related to 
the class 


2. 

Activities 
related to 
study 
support 
system 


3. 

Activities 
related to 
examination 
and learning 


4. 

Activities 
related to 
structured 
notebook 
and report 


5. 

Activities 
related to 
attending 
a class 
and PC 
utilization 


Sum 


I. Attitude related to 
problem-solving and 
sense of fulfillment 


85 


24 


67 


50 


17 


243 


80.6 


27.6 


43.5 


51.7 


39.6 


243 


II. Attitude related to 
thought, creation 
and expression 


23 


31 


9 


40 


19 


122 


40.5 


13.8 


21.8 


26.0 


19.9 


122 


III. Attitude related 
to communication 


66 


6 


8 


30 


9 


119 


39.5 


13.5 


21.3 


25.3 


19.4 


119 


IV. Attitude related 
to computer 


13 


3 


17 


0 


47 


80 


26.5 


9.1 


14.3 


17.0 


13.0 


80 


Sum 


187 


64 


101 


120 


92 


564 


187 


64 


101 


120 


92 


564 




Adjusted residual 


Result of significant test 


I. Attitude related to 
problem-solving and 
sense of fulfillment 


0.8 


-1.0 


5.2 


-0.4 


-5.2 






*** 






II. Attitude related to 
thought, creation 
and expression 


-3.8 


5.5 


-3.4 


3.5 


-0.2 




*** 




*** 




HI. Attitude related 
to communication 


5.8 


-2.4 


-3.6 


1.2 


-2.9 


*** 










IV. Attitude related 
to computer 


-3.5 


-2.3 


0.8 


-5.0 


11.1 










*** 



Note. ***p< 0.001. 



The group 1 “Activities question and learning related to the class” is significantly useful for improving the 
group III “Attitude related to communication”. The group 2 “Activities related to study support system” is 
significantly useful for improving the group II “Attitude related to thought, creation and expression”. The group 3 
“Activities related to examination and learning” is significantly useful for improving the group I “Attitude related to 
problems-solving and sense of fulfillment”. The group 5 “Activities related to attending a class and PC utilization” 
is significantly useful for improving the group IV “Attitude related to computer”. It seems to be understood that it is 
important to cooperate by learning with a PC and by reviewing and evaluating the planned system. 
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Table 7 



X 2 Test and Residual Analysis About Clusters of Attitude and Activities in 2007 





Observed frequency 


Observed frequency 


Clusters of attitude 
and activities 


1. 

Activities 

question 

and 

learning 
related to 
the class 


2. 

Activities 
related to 
study 
support 
system 


3. 

Activities 
related to 
examination 
and learning 


4. 

Activities 
related to 
structured 
notebook 
and report 


5. 

Activities 
related to 
attend a 
class and 
PC 

utilization 


Sum 


1. 

Activities 

question 

and 

learning 
related to 
the class 


2. 

Activities 
related 
to study 
support 
system 


3. 

Activities 
related to 
examination 
and learning 


4. 

Activities 
related to 
structured 
notebook 
and report 


5. 

Activities 
related to 
attend a 
class and 
PC 

utilization 


Sum 


I. Attitude related to 
problem-solving and 
sense of fulfillment 


38 


20 


33 


27 


9 


127 


40.9 


22.6 


21.2 


23.3 


19.0 


127 


II. Attitude related to 
thought, creation 
and expression 


16 


32 


5 


18 


6 


77 


24.8 


13.7 


12.8 


14.1 


11.5 


77 


III. Attitude related 
to communication 


42 


7 


5 


20 


20 


94 


30.3 


16.7 


15.7 


17.3 


14.1 


94 


IV. Attitude related 
to computer 


18 


4 


16 


0 


18 


56 


18.0 


10.0 


9.3 


10.3 


8.4 


56 


Sum 


114 


63 


59 


65 


53 


354 


114 


63 


59 


65 


53 


354 




Adjusted residual 


Result of significant test 


I. Attitude related to 
problem-solving and 
sense of fulfillment 


-0.7 


-0.8 


3.5 


1.1 


-3.1 






*** 






n. Attitude related to 
thought, creation 
and expression 


-2.4 


6.2 


-2.7 


1.3 


-2.0 




*** 








HI. Attitude related 
to communication 


3.0 


-3.1 


-3.4 


0.9 


2.0 


** 








* 


IV. Attitude related 
to computer 


0.0 


-2.3 


2.6 


-3.9 


3.9 






** 




*** 



Notes. *** p < 0.001; ** p < 0.01; * p < 0.05. 



Activities in 2005 and 2007 worked differently for improving the attitude as follows. The group 3 of 
activities is significantly useful for improving the group IV “Attitude related to computer” in 2007. The group 4 
“Activities related to the notebook and report” is significantly useful to improve the group II “Attitude related 
to thought, creation and expression” in 2005. The group 5 of activities is significantly useful for improving the 
group III “Attitude related to communication” in 2007. 

In respect to each cluster about the activities, the % test was conducted about the frequency of four attitude 
clusters in both years to know whether there was significantly difference between attitudes in both years. 
Deflection of the frequency was significant for only the group 1 of the activity cluster (x 2 ( 4 ) = 8.3, p < 0.05). It 
was not recognized significantly for the other four activity clusters. Residual analysis was conducted about the 
group 1. The group I “attitude related to problem-solving and sense of fulfillment” was much in 2005. The group 
IV “attitude related to computer” was much in 2007. “Activities related to question and learning in respect to the 
class” means to help improving “attitude related to problem-solving and sense of fulfillment” in a lecture-centered 
class in 2005 and “attitude related to computer” in the learning-centered class by e-leaming in 2007. 

Change in the Peer Assessment for Planning the Study Support System 

Students reviewed each peer’s reports of the planned study support system for 15 items shown in Table 8 
during class. The results were used for improving the reports. Students rated their reports on a downward 
5-point scale from 5 to 1: 5, very good; 4, good; 3, average; 2, not good; and 1, poor. The information was 
analyzed in the following way. 
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Table 8 



Rating the Planned Study Support System 



Items 




2005 




2007 


T-test 


m 


SD 


m 


SD 


t Results 


(1) Appropriateness of learning contents 


3.2 


0.7 


3.6 


0.9 


1.6 


(2) Appropriateness of learning function 


3.2 


0.7 


3.4 


0.8 


0.5 


(3) Appropriateness of screen structure 


3.6 


0.8 


3.3 


0.9 


0.7 


(4) Appropriateness of knowledge base of teaching materials 


3.2 


0.7 


3.4 


1.0 


0.5 


(5) Appropriateness of instructional strategy 


3.3 


0.9 


3.1 


1.1 


0.4 


(6) Appropriateness of inference engine 


2.8 


1.4 


2.9 


1.2 


0.2 


(7) Appropriateness student model 


3.1 


0.7 


2.8 


1.3 


0.7 


(8) Appropriateness learning method 


3.4 


1.0 


2.8 


1.3 


1.6 


(9) Appropriateness of the method to evaluate an answer 


3.2 


0.9 


2.9 


1.2 


0.9 


(10) Appropriateness of the method to diagnose a bug 


3.1 


1.1 


2.6 


1.3 


1.2 


(11) Appropriateness of system configuration 


3.3 


0.8 


3.0 


1.1 


0.9 


(12) Explanation using correct expressions 


3.3 


0.7 


3.4 


0.9 


0.2 


(13) Appropriateness of the quantity of teaching materials 


3.3 


0.7 


3.3 


1.0 


0.1 


(14) Plainness of content of teaching material 


3.4 


0.7 


3.3 


1.0 


0.4 


(15) Presence of originality 


3.3 


0.7 


3.3 


0.9 


0.2 


Average 


3.2 


0.6 


3.1 


0.3 


0.6 



The numbers of students who submitted their report in 2005 and 2007 were 11 and 11, respectively. The 
numbers of students who had submitted an evaluation sheet were 13 and 22, respectively. A significant 
difference test was conducted among the average values of all 15 items in 2005 and 2007. No significant 
difference was recognized at a 5% significance level as shown in the bottom of Table 6. This means that the 
average rating value in both 2005 and 2007 is the same around as a whole. 

This means that the contents of the teaching material do not become too easy to understand, and in 
addition, it is difficult to judge whether the learned contents are appropriate. We will consider and study the 
educational method for these problems in the future. 

A student wrote a reflection and on how to better improve their writing at the end of the peer assessment. 
The following is typical examples of those submitting descriptions: “This problem is fascinating”; “It is 
unfortunate that the screen is cut off’; “When increasing the knowledge base of the teaching materials, I feel 
that the reports become better”; “I think it may be easy to understand a screen which indicates a problem”; “I 
feel that the description written after the inference engine increased”; and “I consider this to be a better 
knowledge model now”. Students rated peer reports each other and gave praise and advice mutually. These 
activities by the students brought about better interaction as explained above. 

Conclusion 

By comparing the outside-the-classroom blended class with the inside -the-classroom blended class, the 
following was found: 

(1) The students in 2007 scored higher in average points in the small tests. In the understanding state, the 
students using e-leaming in the class could solve exercises in a shorter amount of time, since they filled out the 
structured notebook in e-learning; 
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(2) The results of the questionnaire for the degree of knowledge for technical terms showed that it was 
mostly the same for the understanding of the technical terms in both kinds of the blended classes. Difference in 
recognition for the degree of knowledge was almost not recognized; 

(3) The questionnaire for the abilities and attitude showed that difference in the level of the abilities and 
attitude between the two blended classes was almost not recognized wholly; 

(4) The attitude raised by activities is almost common in both the blended classes. Utilizing a computer in 
each class brought to improving comparability more consciousness than not utilizing it. 

Since a computer is used in a learning-centered class by e-leaming, attitude related to computer is raised. 
From these results, it can be seen that there was not much significant difference between the two blended 
classes. This suggests that one of both blended classes could be used according to the situation of students or 
the environment of PC. 

We hope to study the role of respective media forms and the artifice of appropriate instructional design by 
changing the media, which will be used for a blended class in the future. 

References 

Adachi, K. (2007). Analysis of the classification of the learners’ activities in blended learning. Japan Journal of Educational 
Technology , 3( 1), 29-40. (in Japanese) 

Arakawa, M., Ueki, Y., & Fuyuki, M. (2004). Spontaneous learning activation spiral education method utilizing Web-based 
coordinated education activation system CEAS. Japan Journal of Educational Technology , 28(4), 311-321. (in Japanese) 
Bersin, J. (2004). The blended learning book: Best practices, proven methodologies, and lessons learned. Pfeiffer, USA: San Francisco. 
Bonk, C. J., & Graham, C. R. (Eds.). (2006). The handbook of blended learning. Pfeiffer, USA: San Francisco. 

Horita, T., Murakami, M., & Morishita, S. (2003). Analysis of students’ records in the university class adopting e-leaming from a 
view point of class improvement. Japan Journal of Educational Technology, 27(Suppl.), 145-148. (in Japanese) 

Jung, I., Kubota, K., & Suzuki, K. (2008). Instructional design by most suitable model. Tokyo Denki University Publication 
Office, Japan: Tokyo, (in Japanese) 

Kitazawa, T., Nagai, M., & Ueno, A. (2008). Effect of e-leaming system in the blended learning environments: Exploring the 
relationship between motivational beliefs and self-regulated learning strategies. Japan Journal of Educational Technology, 
32(3), 305-314. (in Japanese) 

Miyaji, I., & Yoshida, K. (2005a). Change of recognition in technical term in class blended lecture and e-leaming. Proceedings of 
30th Annual Conference of Japanese Society for Information and Systems in Education, pp. 367-368. (in Japanese) 

Miyaji, I., & Yoshida, K. (2005b). Practice and learning effect of education by blending of lecture and e-leaming. Transactions of 
Japanese Society for Information and Systems in Education, 22(4), 230-239. (in Japanese) 

Miyaji, I., Yoshida, K., & Namse, Y. (2007). Effects of blending e-leaming and lectures utilizing a stmctured notebook. 

Transactions of Japanese Society for Information and Systems in Education, 24(2), 208-215. (in Japanese) 

Miyaji, I. (Ed.). (2009). Toward blended learning from e-learning. Kyoritu-Shuppan, Japan: Tokyo. 

Mochizuki, T. et al. (2003). Analysis of student evaluation of e-leaming programs cooperated with campus classrooms. 

Transactions of Japanese Society for Information and Systems in Education, 20(2), 132-142. (in Japanese) 

Nakamura, Y., Kojima, H., & Kikugawa, M. (2004). Measurement of basic knowledge about information technology using 
questionnaire on computer technical terms, and its application to evaluation of information education. Information 
Processing Society of Japan Journal, 45(4), 1222-1231. (in Japanese) 

Nishimori, T. et al. (2003). Development of the e-leaming system cooperating with lessons in classrooms of higher education. 

Japan Journal of Educational Technology, 27(Suppl.), 9-12. (in Japanese) 

Saito, T., & Kim, S. (2009). A meta-analysis on e-leaming effectiveness in higher education. Japan Journal of Educational 
Technology, 32(4), 339-350. (in Japanese) 

Thome, K. (2003). Blended learning: How to integrate online & traditional learning. Kogan Page, UK: London. 




